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ABSTRACT ' ^ - . . . " ' 

Evaluation training materials developed by^federally 
funded research and .development laboratories lie re surveyed, analyzed,, 

, and 'described. This project included the collection and analysis, of 
training materials, and the development of The Consuo.er's Guide to 
Evaluation Training Materials. This final report describes the ' 
rationale r f)urposer and activities .of the project and the consumer's ' 
guide, and includes^ sample of the product descriptions included in 

^the- guide. -The anajjrsis of the eV'aluatj.on training products and a 
description of^the relarfcxon ships* between these products are also 
discussed. A^list of titles evaluated in the guide, an index to these 
tr,aining m^iteria'ls, arid an analysis of the trainin'g materials 
pftDduce^l by each research and development laboratory are appended. 
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ABSTRACT 

The Analysis and Reporting of Evaluation Training Materials 

I.. 

Project was condiifcted to survey, analyze, and, describe evaluation 
training inaterials developed by federally funded R&D Laboratories 
and Centers, * - ' 

Its' work fell into thfee . phases. The first, Desigp of an 
information System, includecj the identification an^ collection of 
materials. During the second, Information Analysis and "Develop- 
ment, a conceptual framework wa$ developed t^ use in product 
analysis,: and was appliecj to the materials collected. Product ^ 
descriptions were preVa^ed, alternative configurations of use 

• , t K * 

were described, * and sets of correspon^fling training' materials 
identified^ ' This .phase had two major outcoi^es: The Consumer' s 
Guide to Evaluation Training Materials , which includes descrip- 
tions- of 38 products" and an index; and an dnalysis of content,- 
approaches, and relationships among products, both of which are 
described Tn this report • The third^^hase consisted of delivery 
to NIE of the revised Guide , this Final Report ^ and a display of 
materials. 

< 1 

The Final Report includes a description of the project's 

rationale, purpose and program, a description bf the > Consumer ' s , 

Guide with the Index and sample product descriptions, ^nd ' 

a report on the analysis of the training materials included in 
the Guide* - ' . , ' 
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4 
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CHAPTER CNE: PATICNALE, PURPOSE * 

AND PRXRAM OF IHE PKi^CT , . 



Duri]^^ the past decade the pressure on ^educators to ijiprove the 

effectiveness of education, to assure quality and to be accountable for the 

useyfxiLness of every dollar spent has increased dramatically. In r^ponse to- 

- this pressure , ' school people are actively seeking ways to ^f ind out vAiat 

education should be achieving, v4iat it is achieving in fact and v^re change 

is necessary. This chapter elaborates upon the need for a convenient, 

ft 

corprehensive and up-to-date sourcfe .of inf brrnation abQUt evaluation 'training 
materials, end explains the objectives' and program of the project that was 
undertaken by Far West Laboratory in an attertpt to. meet that need. 



I. The Need ' 

The traditionkl approach to evaluation^ vdiich focuses on the collection 
and analysis of data, is not sufficient to meet the needs discussed above. 
Father than enlightening anyone, the variety of new evaluation philosophies 
and approaches often adds to uncertainty. .Which approach should one choose 

\ 

\ and hcW .should that approach be mastered? 

\ / , - • 

• Eii^erg-ing-iieeds and perceptions have Been only slovly reflected by 
tra^Jiing programs in higher education (the traditional" source of educatiohal 
training). Furthermore, evaluation, ccnpetence is -often desired by people for 
v^tom it is not; convenient to go back to school. ^ I 

As these needs have beccjne apparent, a variety; of organizations, from 
• educational' R&D Laboratories ' and Centers to university departments of 



education and cottneroial publishers, have seized upbn cne aspect or another 

of' the problem and developed nate^ajB and'' progranjs intended to he\p solve it. 

The result' is an array of products of great variety addrtessing the need' for 

training in evaluation skills. 

Because of the plethora of material the educational manager is faced 

« 

with a number of problems: • ' . ' \ - ' 

• he or she often does. not know what" evaluation training materials are . 
. available; r ' ' • ^ 

• product* specifications often do not coxmunic^te exactly" what the 
f>rdduct is, intended to do; < ' ' . , 

• .product descriptions do not always make cleai> their scope and 
requirennents in terms of tijne, money,' and conmitment; 

' • product^ advertising is often the onlysoiorce of information on how 
^ successful the product is, ^d under what conditions. 
Furthermore , those who are responsible for making decisions about what* kincfe 
of evaluation |:raining developm^ should be supported have no systematic and 
conprehensiv^'infoniHtiai that will tell them which. areas are not covered and 
\diich areas are dealt with and to vhat extent. . • 

There is an undeniable need for convenient , coBmprehepsj.ve and accurate 
sources of informatioi . about evaluation training materials. Although * 
catalogues and resougce lists exist in which many of these materials are ' . 
described, there is still no source that will help a prospecti^^ user solv^ 
all the problems listed above.- ' ✓ 
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II. Statanent of Purpose ' \ ^ 

m 

The project described in this report 'attenpted to met the needs of both^ 
users ahd funaeVs of develcping evaluatioi;i training materials. The NIE staff . 
caisidered this vork essential to the needs of >±ie Basic Skills Division of 
NIE in general, and in particular, to the needs of ^isasuremsnt and riethodology 
for infonration that would help it coordinate development of evaluation 
materials by Laboratories and Centers and improve the methodological quality 
of future evaluaticn t2;^ining products. . \ \ * * • 

The project vfech Far West Laboratory conducted had as its goal a survey,^ 
analysis and description of evaluation training materials developed by 
Mucational Laboratories and RSD Centers. This survey and analysis provided 
ir formation that can be used by educators in making decisions about using 
these materials and by NIE staff in considering the need to continue or 
-promote the development of new materials in this area. \ 

Since the develc^^noit of these ma'fce^ials has been si^ported primarily b^ 
public funds it is particularly iir^xDrtant that knowledge about than be made 
available to the students and professionals in education for v^on they were 
intended. One result of the current project should be to assist in disseminating 
information about products already paid for by the tax--payer. 

c 

For the same reason, if future' spending in this .area is to be cost- . 
effective, ne^ projects chosen for support should be aimed at areas within 
evaluation training that are inadequately covered now. 



Ill* * The Program of the Project , • 

The program of the project can be reported in the following thitee* phases: 
Phase 1; Designing an- In forma ticn Systan . This phase, v^ich occupied 
the spring of 1976, was devoted to designing a system for collecting and 

'analyzing evaluation training materials. It included working with NlE staff 
to develqp ^ frane of reference for the collection of such information; usiSiq 
this infonnatia;]i to identify appropriate materials for consideration; 
surveying other sources of inforrtBtion for n^mes of appropriate materials; 

'rev\^wing literature that described evaluation coipetenc^^ and specified 
ascertaining user groups' perceptions of training requirements* and corpetence 
areas; and synthesizing the information gleaned frOm these activities. t(^ 
develcp a list of materials to be analyzed and critSeria^for analysis. 

Phase 2; Information Analysis and Oevelcpnaent . The three main tasks 
performed during ,this phase were the collecting and analyzing of the - 
materials and information about their use; defining aijd describing alternative 

^ 

configurations of use and sets of training nateijials that correspond to each 
other; developing an organized description of the materials reviewed and 
analyzed; and reporting 'this infopfation. 

Oyer one huhdred trajjiing products and programs were considered and 
reviewed ♦ Of these descriptiais , thirty^ight products or programs were 
prepared to go into a^ Gonsumar's Guide to Evaluation Training Materials 
These descriptiais were reviewed by developers, consultants and NJE staff and 
^formation en the history and evaluation of each product were collected. 
A ^iample 'description is included in T^pendix A. ' An index for classifying ancT 
analyzing products was' also developed". Tli^' index is included in /^pendix B. 
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A c»nceptual map was also develc^sed as fraitfework for presenting an- . 
analysis of content, approaches and relationships samong products. Chapter . 
Three deals with this.. activity, 

ArtivitUes in 'this phase included further revision anc3 review of product 
descriptions, the additicn of evaluative carmen ts, and 'additional deyelopinent 
and refining of 'the analytical tables, 

Ph ^e 3; Delivery to NIE of a-^Consumer ' s Guide and Report and 

Reccmnsndaticns , * In this phase, the project delivered to NIE copies of the 

CDnsuiTBr's Gaide and h technical report that included reconmendations foi" 
- — f , t 

criteria and procedures for determining the quality of training materials as 
well as for deciding which areas merit continued develofSiBnt and \^Mch have 
been insufficiently covered ard thus need new development, ' 

A display of training products included in* the Guide was^also prepared 
and sent to NIE. 
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CHAPTER TVK): liSCRIPTICN OF THE 
CONSUMER'S GUIEE 



. ^ This ch^ter reports upon the content and format of the Consumer's Guide 

after identifying the audience tAat would make best use of the information 

. covered. Several criteria. for the inclusion of 'materials are presented; 

' follcwoJ by an expl^atibn of the types of material' included, the method of' ' 
.» . ^ ' . . . * 

^resentatioi / sources of informaticn and suggested j^ages, 

1. The Target Audience 

The Guide was written, to help all those vto need inf ormtion 6n and 
'training in sane aspect of evaluation. Although th^ primary users of the 
Guide will professionals who seek materials to train other pecple in the 

various dimensions of educational evaluation, other users will probably 

'. * * ' 

include school administrators, other school -staff numbers participating^ in an 

evaluation or coordinating evaluation efforts, school ^ connunity people 

^ serving on evaluation canmittees, "program or product developers who need to 

» 

test their work and entry professionals preparing for careers in education, 

2. Criteria for the Inclusion of Materials ' 

^ B^use NIE want^ information on products developed under its own, 
" spmsorship before exploring products developed by others, the first , 

a 

.'criterion for inclusion. was that the product presented had been developed by 
a laboratory or R&D. center. -Jt is anticipatedn that in an extension of this ' 
project, Tnatetials developed by^ university departments of education, coimercjigl 
publisj^ers, independent developers and others will also be covered. 
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, The second requiratient was, that sorfe part'of the product presented dea^. 

with an ^aspect of the ^evaluation process or a skill used the^rein. Most ^ 

. materials address either the tasks involved in a given stage ':^f evaluation^ 

such ^ sumtBtive o^-, outcaT>e evaluation, or^a skil^l, sao^ as evaluation ^• 

planning, that can be applied at any one of several points duting an 

* • 

evaluation effort. The materials p^-esented are elso limited to those that . 

/ ' * 

deal with sane aspect of the educational p;?oces$ (curriculum, 'goals', programs, ' 

etc.)*/ rather than with the assessnnent of personnel involved* in that process.^ 
<' • «• ^ » • 

The third and final criterion Wpis that 'a product '^ust provide , evaluation, . 

training of sorre ^ind, rather than merely exploring, the subject of evaluation, 

proposing an approach or philosophy or reporting findings^ Thus, all the 

naterials presented are specifically oriehted toward users vdio expect to 

participate i,n the evaluation process an(S most provide for user involvement. 

3.. Types of Materials Included . ~ ' ) 

Materials are presented at three hierarchical level^^program, corponent 

^ and iTvodular — depending* on their degree of appropriateness and ^ their 

>. t . 

separability from other materials gr- larger configurations.. ^ ' ^ 
* . *^ • ' i ^ 

• . Included in the program level are groups of ' related products. These 
materials mayfiave various titles ("program," "serie^," "System" or "resources" 
depending on the develc^r involved.'- Products may also exist at the dorponent ^ 
level — a corponent being^ a product that can stand alone. Here, too, titles 
may differ, with •sane developers usj/ng "systems" to refer to mateirials of . 
this type, and others using "series,"^' or presentaj^_J;he^^^^Qonponent as a 
numbered section of the program or bjj simply naming^^em^ Those corponents 
mentioned as part of a program will bd Hescribed in their cwn rright if they 
include evaluation training. Similarly, subdivisions or sections ofi 
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conponents that ^ocus on evaluation training and can' be. used independently as 
nodules itiay also be described independently. > * <, • 

^ /How the Material is Prese nted - , " 

/ — — ' f 

:^ , The jouide consists 6f an introduction, an index to the materials 
inclxxied/.a list oi the materials suimiarized and the surtmaries theneelves/ 
Because the Guide is deeignied to needs of the evaluatprAiser, it 

J^dif f ers from the •NIE..cat$logue in' the f Allowing ways : ! ' . 

• All mterials with evaluaticn content or aspects are included, even if 
evaluation is not their primary focus. 



• Ths nature of evaluation training is described and the approach to 

/ ' * ' 

.evaluation^ somarized.. - . , ^ 



• The Guide ' s index allows product selection according to a number of 
factors/ ranging fifan~~target audience^ to type of ' evaluation needed. 
^ a. * The/ Index 

: / ' • , . . ' 

The ing[ex> is designed to he\p tte user quickly .locate the surrmary (or. 
. ' / • ^ - ' " , ^ 

sunmaries)ydescribing the evali:iation draining materials^ that best suit .hiS' or 

.her needs. Materials are thus classified according to three major character- 

isticg: , intended liisers / foois and type of evaluation. . . ^ *-»Sj 

^•the first heading , intended users , hel^s one^ find products' that should 

be c^sidered if the usetr belongs to or is working with a particular group 

(In^ainef, -administrators, canmonity pe'cple, etc.) . Since many of the 

materials have been de&igned for use by more than one group, ^ the fact tha^ 

a prbduct is not listed under a specified group does not necessarily inply 

that the product would be in^ropri^te. It^merel^ indicate^; that the 

developer did not id^tijfy that group as the intended user. ^ 
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The* second characteristic/ fo cus of mteria lS/ helps one locate prodtJCts 

— — ~~ — ^ — - - - # 

that address one or several specific content areas (curriculum, instrunenta- 
tion, attituies, etcO. that are part of or associated with evaluation • The " 
third heading, type "of evaluatiai / assists the user iif- finding itaterials that* 
address the stage or stages of evaluation that inte^rest him or her (needs 
assessn[>ent , planning, fornetive or sunrnatiye .evalTjlatiC)n) • 
.-^ b* The Surrmaries - * 



Following the index are the product sunmariQs . Thi'S" yersio;;i of ttebaide 



includes 38' products developed by the Center fot Educational Policy Management 
(CEPM) , The Caiter for the Stujy of Evaluation (CSE) , Far-' West Laboratory for 
Educatiaial Research and Developnnent (FVJL) , the Northwest* .Regional Educational 

• • t ^ 

Ijaboratory (NWRL) , Research for Better Schools, Inc/ (RBS) , Southwest Educa- 
tional, Development Laboratory (SEDL) and Sputhwest^Regional Laboratory (SWRL) • 

Sect^^s that :yiclude all products by a given agency are arranged 
alphabetically according to an acronym cot^psed of the product developer's 
initials and a nunnber^ ^hus,CEPM 1 would be the first J.isted product 
'developed by the Goiter for ]§^ucaticHial Policy Jtoagement,. This arrangement 
facilitates the addition of new, proc?ucts and new agencies to the ' Guide without 
having to ^change page numbers. - t " ^ ^ • , 

The sumnaries themselves have a. uniform format to iraKe it easier for the 



user to cotpare th^. The description* of each product app^^j^^sjjnder a number 
of headings that are intended to provide all the infornation a user would need 
ifi order to make a choice. Although non-technical language is 'used wherever 
possible, there are times vrfien only a specialized term can^ convey the desired 
meaning. For this reason, a sample two-page product? sumnnary that explains 
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the kind of »inforTOation' to be expected under ^ each heading, is included. It 
a£)pears here on pp. 12-13. * ^ 

. . 5. .^^urces of Information. 



Information ,about'the materials' in this Guide cams from the agencies 
that •cte\;^8tied them and the materials themselves. No matter v*iere the, 
information originated, ccnpleted, summaries of the material were^sent to the 
developing agencies for. comrtent and revision. Since rthe work was done during 
sthe^svininer and fall of 1976, the materials presented in this Guide are ^ as 
accurate and up-to-d^te as of that time. 

6. Using the Guide 

Although users. "Sf the Gui^ can simply le'af through it to see vdiat types 

of evaluaticn .training materials exist, most readers will probably turn 

straight to the products that are most likely "to meet their needs. This can 

be acccnplishsd by using the index to identify the summaries that would ,be 

most suitable. One can then fim thfe siommaries^-in the Guide . 

*.* , ' '> ' 

^ For-exanple, Suppose the ^superintendent of a medium-sized school district 



wants to"teaGh his -or her own staff to evaluatq, the' junior high school 
oirricxilum because budget limitatic^s preclude hiring an outside professional./' 
He or she would begin by checking the section of the index headed "Intended 

■ ^ . / y ' ' 

Users." After discovering that a large am^/unt of materials are intended for 
use' by school— staff members, he or she would then move down to "Fociis of 
Nfeterials" to find out which of these products are aimed at curriculum 
evaluation and to "Type of Evaluation" to see \Adlich mterials cover formative 
and sumrative evaluation. By this tinive, the superirjt^dent has narrated his 



or her list to a^tialf dozen products that fulfill all three criteria. These 

• ' i 

— , — _ — ( ^ 

are identified" by initials that stand f6r^±he_d©/eloper 's name and numbers 



referring to the -products' placement within each section. Thus, v4ien the 
siperintendent wants to find CSE 5 and 6, he or she turns to the Center for' 

, tjie Study of Evaluatioi section, finds tte fifth and sixth product suimBries 
reads them, notes such information as cost and pro\^isions for use and' then 
goes on to look at EBS 3, SWRL 5 or at \^*iatever other sections have been 

• suggested by cross-checking prodiicts under "intended users," "focus of 
inaterials" and "type of .evaluation-" After corparing the. various product 
summaries and figuring out T^Mch programs are likely to meet his or her 
district's needs, the superintendent can write^ to the address given under 
"Availability" and secure more information about the product or obtain tte ' 
product itseuLf . The next two pages introduce the .schane in which the 
description .of products was developed fot .the Guide. 
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PRODUCT TITLE C^IPLS if any) . ' ' ' , j 



(name of larg^ program product -is a part of, if any) 



INTENDED USEIS . ^ 

The target audience/ or people, at v^icm the product is niore specifically aimed, 
and for vhon it will presumably work best. 

TYPE OF EVjP^LUATICN ADDRESSED / ^* ' ^ 

JTbe stage of 'evaluation 'SM/or evaluation skills for v^toch training is being 
provided in this product. The main stages of evaluation vMch 'may be listed 

are: ' . ^ • * ^ ' ' 

a) pireformat^iye -'needs assessment, context assesarient, program planning/ 

and any other, activities tijat take place before or 
early in the deyelcpiien^of a product or program, 

b) formative - evaliiltion that take^ place dxiring product or program 
development or that^"^ is intended to inprove a progr^m ^ 
that is currently"^ in operation.. 



<3 



^ c) sunmative - evaluation tj^t takes place aft®^;.,^a product or program 

has been c0TTplet^ and put' into ope^fetion, whose purpose 
* ' is to det^'rmine its wrth^^-and- provide information for 
V depision on whether to ^pontinue or terminate it . 

The main skill areas tfial: Jtiay be listed are: , '''"^ * * . 

a) technical ^ instrument development, research design, data analysis. 



b) interface ^ interpersonal relations, '^omnnunication of evaluation 
/ . . results. 



c) adniixustrative - planning, coordinating, or monitoring the e\;^lqation 
- process . . • 



0 '■ ■ 

APPIOCH TO EVALUATICN . * 

The ba^c assiiT5)tions about the nature , and purpose d£ evaluation upon vMch the. 
develcpe^^^of this product have based thdir work. ^ ' \ 

MAIN EMPHASIS • . . 

y 

A brief sumnary of product content, which should indicate, among other thirds, 
what proportiai of the materials are devoted to evaluation trailing,. ' *^ • 

goals" MD OBJECTIVES " ^ ' ■ 

What the materials are e^q^ected to teach aj'^er to do. 

MAIN ACTIVITIES 

What sorts of things the user will .be doing to enable him to achieve tfife objectives — 
this heading Would include specific learning tasks or thg instructional approach used. 



PROVISICNS FOR USE <. ^ 

A. P6iTttat ; Possibilities include workshop, self -instructional, college 

course > on-the-job training, etc. * • 

B. Personnel ; This section indicates v^^lo, in addition to the learner 

himself, or herself, must be present for learning to take place. 

C. ' Product Ccnponents ; ^he number and nature and titles of the .physical 

materials involved, incluiing handbooks, instructor's guides, 
worksheets, and any other required mat^ials. 

D. Fesourpes- ; Any other items; aspects df the setting, etc^ , which are 

necessary to use the materials. -Possibilities here vrc)uld include 
* ' a ire^tin^ roan of a certain size, copying facilities, etc. , , 

E. Time Span ; Hov long it takes the' average user to ccitplete the materials. 

F. Conditicns of Use ; Gondij&Lons v^ch should be* present in order for the 

, materials to bemused effectively, particularly any 5aiTmitment or 
attitudes which should be present in users or tliose responsible for 
the training. - • ' 

ADAPTABILITY - . , _ 

5 . ^ 

A brief indication of any environments or user gro\:ps other than those for 
v^ich the materials were originally intended with which they can-^'be success- 
fully lised. ' • , 



t 



EELATED P]R0D()^3rS * , ^ 
^ 

Other evaluatioi training materials that are sijTtM.ar or related to this one . 

COST ■• . 

How iTUch it co^ts for the materials themselves or for iirplementation of 
a program of which they are a part. ^ v 

AVAILABILITY 

Where* to worite to obtain the materials or information' a)6out .them. 

HISTDRy Alk) EVALUATICN " - \ 

A brief account of hov the materials were tested or otherwise evaluated, with 
the results. This information may not be available for all products. 
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A note by: the developers of this Guide vMch includes any jlnforlnnation not 
covered under any othea; heading, a general assessirent of the product's useful- 
ness , ^d arf/ quali|icatiais ,t^^that usefulness which scrutiny of the product 



may. have revealed;. * ^ 7^ 
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THfe:: A REPORT CN HE ANALYSIS OP 
'TRAINING mTERIALS INCLUEED IN THE GUIDE 



Supported by the National Institute of Education, the Far VJest Laboratory 

oonduc±ed a survey and analysis of Evaluation training materials developed by 

educational, laboratories and R&D centers with the goal of producing a oonsuner 

guide with descriptions of individual products for potential users and a 

tedinical paper for NIE to aid decisions about desirable directions for future 

product developn^Hnt . The consumer guide has, gone throu^ cycles of review and 

revision and has been sulmLtted to NIE and selected e5<pert reviewers for .furtlier 

isfiLtiquing prior to \final revisions. Even in. its "final" form, however, it is 

yiewea as an open system that cah be added to as new evaluation training 

prodixits becoTve available or if the scope of products included in our anal^ysis ' s 

broadens with further funding frctn NIE. * This' Technical Report prov^es: . 

(Ih a cgnc^tual framework of the evaluation process andr the educational" 

• i * * 

dormn in which educational evaluation Ccui be performed;* [2) ^a classification 
of the evaluation training materials analyzed in the "course of the proje<i± in\ 
terms of gaps and overlaps in coverage; and "(3) a set of sample configurations 
of uSe of specific training materials to meet certain configurations of 
evaluati<!xi training needs^. ^ , ; ' 
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!• A Conceptual Franvework of Educational Evaluation 



Irj order to* provide l^IE witii informtiai regarding the 'areas of educational 



evaluaticXi v^ch are and are not covered by the evaluation training materials 
proiuoed by educaticnal laboratories and R&D Centers, we found it useful to 
dbvelc^ a "conceptual nap" (Figures l.and 2) characteriz:^ areas of evaluatioi 
and their interrelatiaiships. Since the vfas intended to represent the 

' universe of educatiaial evaluaticn on v^ch evaluation training materials could 
be located, we sought 'a variety of sources of information about concepts and 

^aspects of evaluatioi fron wiiich to develop it. The' most helpful sources were: 
the literature en evaluation philosophies/nrodsls and methods, our am extensive 
e^qjerience in conceptualizing and conducting evaluations in educati^al contexts, 
and our prelijnainary analyst of the existing R&D ev^uation training materials • 
Vie found it loseful to distingviish two major dimsnsiois or layers of the map: 
'{1) the evaluatioi process dimension (Figure 1) > and (2) ;fehe^educational 
dotiain in response tq v^ch evaluatiois are coiceived and conducted (Figiice 2) . • 
These two dimensions permit analysis and classification of materials f^^om 
a broad systems perspective and make possible the development of detailed 
information for decision-making about , further directions for materials develop- 

Ttent. In additioi, this coiceptual map of the "evaluation universe" could 

facilitate understanding of the relatioiships' among ^the many diverse definitions 
or coicepts of evaluatioi now prevalent and could suggest possible synthesis or 
alternative conceptions. Hopefully, then, the map vdJLi£.£^/^e .interesting and 
usefxiL in organizing, clarifying, and extending* the concepts of evaluation as ^ 
v^ll as in providing, a framework within v*iich to describe and . analyze existing 
evaluatioi training materials. ^ ' . ' 

' 15 ' • . , 



Below we present a description of the elanents and structure of the rn^p* 
We will then descyribe hov the map was used to analyze and classify the ♦ 
training naterials in order to demonstrate gaps and overlaps in coverage of _ 
the areas and types of educatiaial evaluaticri. . ' ' 



V I 

A. , Evaluation Proces s Dimensiai of the Gohceptual Map \ _ 

t/^ — :r - ■ ■ ■ " Z : 

i • ' • 

Stere is considerable disagreenfent as to what evatoaticn is and* how it - 
should be carried out. However, most evaluators would probably ^ree that_ — ' 
evaluation could be described as the gathering of informatioi about phenonena 
(programs, products, policy "1 etc.) in order to make or help make decisions or 
judgnents about those phenonsna. Gcnoepts' as divergent as "^valuation as- 
juc^ment," "evaluation as information management ori^ted toward generati ng . 
data for dscisicn-makers , ^' and "evaluation as ocnparison of performance with 
i,behayioral objectives" can be fit irito"this broad corioept of. evaluation. The 
' evaluation process" dimension of. the conceptual map (Figure 1) represents this 
broad concept. Tte activities numbered IlTVI^on the m^ could be called the 
"technical" aspect? of the evaluation process. Ttost models of 'evaluation 
include these activities as part of the role of an evaluator althoi^h sone 
models (e.g., the.'dscision-facilitative model) would. consider the making o^ 
-judgitents or decisions to be the responsibility of the administ^ratpr rather 
'th^ of the' evaluator. , Tte nurrbeES II-VI inp'ly a likely sequence of those , 
' a::tivities: - (fesign -of -the evaluation, collection of data, analysis and 
' inter^etaticn of data, the making of judgments and decisions based on the 
data analysis, and the reportii^ of ccnclusions, reccnm^dations , or decisions 
darivBd frdm the data analysis. Design of the evaluation (II) refers to the 
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— FIGURE 1. EVALCKTICN PRDCESS DIMENSION 
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plannir^ of the activities to 'be carried out in the evaluaticn. Though "design" 
,is most ocmtoily used to refer to ^ta generatibn activities (e.g. , sanpling ^ * 
procedures > e^q^eri^mer^tal design such as use of pre--post measures) > it can J * 
involve ther^lanniM Of any of the evaluation activities (e,g,, hav the final 
evaluaticSi report will be s1;:ructured. and presented) ^ Coliection qf data (III) 



iiicludes instrument 



slc^xnent or selection (e.g., choice of test format, 



nontative vs. criterilontreferenoed testing, simulation exercises) and evaluation 
procedures' .(e.g., conrent analysis, field testing). Analysis and interpretation 
of data (IV) can .involve .statistical and ncn-statistical me.thods for suninarizing 



data and/or for drawing i^erences from it. ^udgment/decision-making (V) 
' involves specification jand use of crij^eria^and standards for makirig judgments 
or decisions based on collected and analyzed information, Reporting or 
dissgninating (VI) involves jcarniunication (by sUch devices ^ evaluation 
reports, v^orkshcps, news leileases) of, sumnaries, conclusions, reccrnmendations , 
^ or judgments to various "ciohsumers" te.g. developers, users, funders) . 

Figure 1 yicates tticit thes^-tech^cal evaluation activities are preceded 
by another activity — focu^jng the evaluation (I) . Before nethods of caiducting 



the evaluation can be planned or carried out, it is necessary to determine 



V the purpose or purposes of 



evaluaticn trying to provide aftswers for, wh^t kinds of decisions are. going 



to be made cn the basis of 



is conducted for different 



the evaluaticn; \<fiat kinds of questions is the 



the evaluaticn findings, what kinis of infonrfeition 



are needed. Figiore 1 indicates thiree major types of evaluation, each of v^ch 



purp6ses'^^&'""aiiswer different kinds pf questions 



and to facilitate dif f erenip kinds of decisions . Preformative evaluation (A) 
is oriente^. to decisions oj: judgments about the needs for the program ^(or - 
product or vdiatever phencmenai is to tpe evaluated) and about hov the program 

■ . r • ■ 
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^ shoiild be designed (including go^ls and ob^eptiyes]^ in i^rdelj^ to iteet those 
needs. Prefounative evaluation^ i!? dene 'before pjcoducing or ^ developing the 
program. (In this discussion of tl^e purposes or types of evaluation, we will 
speak of evaluating progr^ , but ,tl;iis is only to sirrplify the discussion, 

' 'The - educational ^donaia dinnension of \he map — Figure 2 — indicates many other 
phenonena besi^s programsN^hat cpi^ be the ta2;get of .evaluation.) Formative ' 
evaluation (B) is that' conducted duripg \the developing and iA ori^ted to 
decisions atx3ut making revisions in the program. Summative evaluation (C) 

. ' ' t ' 

IS acconplished after the developing 'ils ccnpleited and is directed tavard 
judgments or decisions of program dispipsitioi ^(e.g. , termination, adoption, 
extension ,^ ejq^ansion) . Evaluaticn of an educapignal' phenoi^on could include 
any or all of these types'. The A, B, -^d C. "finge2::s" of the focus activity 
extending across activities Il-vr indicate that focusing an evaluation not 
only precedes the other evaluation activities, but continues to occur ^ 
thro\:ghout the other activities and interacts with than. For exanple, in . 

analyzing the data previously coB.ected or in reporting conclusions or reccnr- 

^ ' ' ' > , . ' ^ V 

mendations, the purposes of the evaluation and the questions to be addressed 

cor 



most be carefully kept in mind. ^The 



)ntent and/or plxx::edures of design, 



datai^collection, data an^^ysis, decisi::^-making^ and dissanination may differ 
"-substantially depending on the evaluation's purpose. 

* Though Figure 1 indicates ^that part of* the focusing of an evaluation is * 
ascertaining and specifying its genercl^^Jurpose, i.e., determining needs 



(preformative) , revisions (formative) , 



or disposition (summative) ; focusing 



^ • . t \ 

also involves^ determining specific "purposes and types of questions to be answ^ed 

(e.g.,^ hew do users like the program, how.nruch do users learn from the program) 

v^di will enable the general purposes, of the evaluation to, be achieved. Part 



.ERIC 



19. 

24 



of detennihing this specific focus involves ajmrunication with potential u^ers 
. of the evaluation information to identify their -needs. Infeeaspersonal relations 
and ocOTTunicaticai is represented in Figure 1 by interface . ^ interpersonal 
contact to help focus the evaluation is represented by the o^rlap of focus 
and interface . The overly of the interface aspect of evaluation with 
activities I-VI indicates that interface concerns accorpany each pf these 
evaluation activities; the needs/ attitudes, e^^riehce, 'and capabili^ties of - 
the people fran \>^an information is" to be" gathered and of those will use the 
evaluation inf onnation have to .be considered throughout the evaluation process 
in order to conduct the evaluation smoothly and effectively. For example, 
ethic^ considerations of the personal privacy and/or psychDldgical health of 
^ respcxid^nts to questionnaires or tests would be represent^ on the evaluation 
process dimension of the ccnceptual map by the overlap or interface and 
colljcticn of data > The consideration of charlact;!eristics (e^g., level of 
sophistication, attitudes) ' of the' audience for a final evaluation report so 
that vA)en it is j^esented to that auiiqice it \/ill be understood and acted 
upon is represented in Figure 1 by the overlap* of interface and report/ 
dissemination > Even an evaluator's analysis: and interpretation of data , which 
is often performed without irput from potential users of the evalization, could 
(and perh^s should) be influenced by canmunication jvith those users, for 
clarification' or modificaticn of u^er needs may lea^-to revision' or expansion 
<5f data analysis. I>fe perc&ve'the evaluat^o^ pi;ooess, then, as requiring 
interpersonal skills ( interface) as well as'-technical procedures skills* 

Figute 1 also indicates that administration (e.g., scheduling, budgeting, 
monitoring) of the evaliaation occur^ throughout, llegardless of the purpose' . ' 
or typQ of evaluation, all evaluation aetivities must be administered. This 
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is represented in Figure 1 ^hy the overlap of administer evaluation and the 
oti>er evaluation activities. / * ^ , '* • a 

The entire evaluatiai process take? place within and is influenced >y 

^- ' 

institutioial or systan constraints , such as the degree of ecaicmic dependence 

in^pendence the evaluator has from the program being evaluated or the 
P9litical br ecaianic ^iif iuences on what questions c^ be asked ard ^t types 
of answers will be •^6cepted. 

The evaluatioi pitscess dimansiai of the congeptual map is intended .to ^ 
repres^t the interrelations of the fiall range of processes considered by 
diverse theorist^ and practitioners to ccrrpris^ evaluation. Evaluation . 
training materials reflectii^ ' diverse ccncepts of evaluation can be analyzed 
and classified according to theif, location orl this diinension of th^l3^. Such 
classification shDuXd provide NlE with information about the distributiai and 
.deiasity of coverage of process areas' of evaluation by existing evaluation 
training materials. This information should then be useful in making decil'ipns' 
about 'directions for further materials development., ' ' 

There is, havever, a second c^im^ of the coiceptual map VN^ch should' 
be considered in characterizing the distriJxitxon and density of the R&D 
evaluation training itaterials and in deciding cn directions for future 
• developinent of material^ This, second diitension (represented .in Figure 2) is 
' the overall dafnain of education ani its various corponents within which the * , ^ 
evaluatich process operates. Although therei are mafiy'v^ys of conceptualizing, 
the domain of education^ we feel the ^ f ollo/ing ^ran^ork is useful for the 
purposes" of this paper, for it is relatively siitple, yet allows differentiaticn 
between evaluatit* training materials on^tiib basis of the aspec± of education 
being evaluated in those materials. , > . • , 

'. X 
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Figure 2. IXmain of Bducaty.on Dimension 
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B. E)omain of Bducatioi Dimension of the Conceptual Map ' *^ 

The dohain of education (Figure 2) operates as. a coitplex system of 4 

canpcnents that engage in various processes and produce various outcoires'. 

) * 
Tte term processes refers to tip dynamic interactions both within and amopg 

the ccnpcQ^ents of the syston, Ttese ar^ usually expressed or manifested vc\ 

tferms of procedua^es, relationships, or methods • The terJm outcoannes , on the 

other hand, refers to the results c^the interactions of the coirgpnents,. 

Qitcones are usually expressed in terms of achievement, quality, gain, 

ef f ecrtiveness , etc%. One way of 'conceptualizing the components of the 

education domain within .'and among which these processes and outcomes occur is 

as follows :* 

1, Settings (within which the education processes or outcomes occur) 

1.1 Sdiools— includes all formal educative , organizations: public and 

private schools (K-12) , junior colleges, public and private colleges 

and universities, trade ^ and business schools, • • ^ 

* • 

1»2 Professional organizations — includes business -and industrial 

organizations, state or intermediate educational- agencies, R&D centers 
1 and laboratories, professional associations. ^ - 

1.3 Coninunity- Tincludes various neighborhood, civic, social, athletic, 

political groips, clubs, ccimiittees, ,et'c. , organized both 
formally and informally. 

1.4 Family -includes merhbers of a-family unit: parents, children, 
relatives.* . m im ' 



' *The categories given under each se ating/ level , and content are not necessarily 
exhaustive. , f 



Levels of Focxis (i.e.., the level at which JJie educational processes or 

— ? :~ , ^' 

cutcxitBs are considered) 

2.1 Scx:ietal — pertains to educatiaial processes and outoonss as they , 
affect or concern society as a viiole or a societal body that governs , 
influences V or is infi^snced by the educational system. 

2.2 Institutional — pertains to educational, ^^cesses and outccanes as they 
affect or concern an educative institution or organization, either 
piiblic or private, formal, or informal. 

2.3 Instructional — pertains to educational processes and outcomes as they 
affect or concern teacners, 

2.4 learner — pertains to educational processes and outcomes as tliey 
affect or concern learners. 

Oontents (i.e. the aspects of tlie educational processes or outcomes of 
concern) 

3.1 Policy—includes rules, procedures, and priorities \s^<:sa which the 
^ ) 

educative process or the o|perations of an educational institution or, 

v~ — — , 

system are based. 

3.2 (Management) Besources — includes budgets, facilities, staff 
allocations. - \ 

3.3 Programs/curricula — inclujfes instructional strategies and jnethods, 
leaning activities^. 

3.4 Products— includes various knowledge developanent and skill training 
mate rials • 

3.53^ People — includes teac±iers/instructors , pupils/students, progranv/ 
curriculum plannejcs and developers, sdiool administrators, R&D 
specialists, and a variety of ^professional arid 'support personnel 

) involved with an education instit|ation or the eciicative process. 



The relationship between processes and outccros and the settings, levels, 

and contents oorponents is illustrated in Figure ^3-. As is shown,, the donain 

of education .can be thought of 'as a three-dirt^nsional cube ccnposed of 

numerous cells, with each cell caisisting of a particular ccnfcination of 

setting ,^ level, and content, and enccrrpassing both processes and outcores. 

Within any givei? cell, the, evaluaticn process (illustrated in Figure 1) 

--^can be'm^ped, at least theoretically, thus establishing a relationship 

/between the two dimensions of the conceptual- inap, Vfe are saying, then, that 

any of the aspebts of educaticxi represented by the categories and their ^ 
» ^ ^ ** 

, intersections on Figure 2 can be evalua^, and any evaluation training product^ 

can be classified according to the aspects of evaluation it addresses (Figure 1) 
an4 the aspect of education (Figure 2) it uses as the target of the evaluation 
process described. 

Vfe will next present sore exairples of our classification of the R&D 
eyaluation training^ mterials using this mp.in order to illustrate its use 
'^^and to clarify the meanings of its categories • ^ * 



Use'of the -Qaiceptual Framework* 

To illustrate the relationships between^the* R&D evaluation training 
Materials, the process of evaluation, and the larger context of education, 
the training materials were analyzed in terms of their applicability to the . 
evaluation prooess and their location -within 'the donain of -education. 



^It will be noted that no mention i-s made of .the desigri (II on Figiire 1) 
aspect of evaltiation either ih the text or tables of our classification of 
training materials. In the version of the -cdioeptual map we used to classify 
the training materials, design was included as part of data collection . After 
we decided to view design mere broadly as tte planning process applicable to 
any of the other evaluation activities and include -^t as a separate category 
on the- ocnoeptual msp, we reanalyzed the tradaiing materials for their coverage 

* , - Continued footnote, next page 
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^ For exanple, the Progress Evaluation Workshc^ developed by the Center for 
the ^tudy of Evaluation at UCLA ("CSE 6" in the Gonsune r Guide I yas classified 
cai the ev^uaticxi process dimension of the ccaiceptual map as "1, B, iii, iv, VI" 
indicating that it otphasizes fcpcusing (I), collecting; (HI) , analyzing (IV) 
and deporting (VI) aspects of ' evaluation done for formative purposes. This 
clas^if icatLon indicates that we did not feel this product addressed pref^orrna^- 
tive -or sunmative evaluatioi, administration of e^^aliiation, analysis of 
cxEistraints on the conduct of the eyalioation, or the judgment pr interface 
(relations with people) aspects of eAluation, This classification \/as derived, 
in-p^, frou the specified goals and ci)jectives of the product (e.g., 
"participants will be able to specify the purposes. • • of the progress 
evaluation/ (and) . . . will be able *to make decisions about the kinds of oroqress 
inform^cA to gather and the methods to be used in collecting, organising, 
analyzing, and reporting that information") . Since this proddct focused on 
evaluation of educational prograns (c) in schools (A) , according to student (e) 
attairment of <±>jectives (i.e., learner level focus~4) , it was classified as 
"A, 4^ c, e" on. the domain of education dimension of the conceptioal'map. 

Sane products seen to address many aspects csr^valuation and/or education 
v^iile others cover a narrowfer range. Seme products present orientations or 
general descriptions about types of evaluation whil? othe:^ pjfovi'de step*by-step 



Can't,,. 

of design in this sense- Although we felt that most of ^the materials, addressing 
the oonduct of the various evaluation activities could be used to train ' 
evaluators to plan or design these same activities, we' found no materials 
that -provided training specific to the design or plannifig (as distinct from 
the actual conduct) of evaluation: For this reason, the desigrTcategory does 
not appear, in the tables or text. SCxve people may feel that this is a serious 
(deficiency in existing training materials; others may agree thkt as long as 
the I^eed for design is understood, training materials which focus on ho^v to 
conduct an evaluation could also be used b^ an evalioator- to learn how to 
design or plan that evaluation. o ^ • ^ 
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hcw-to txaining, CXir classification' distnSigm.'shes between these tvvo types 

of materials r Several products provide tr^ning in evaluaticn methods used 

\» " "> 

for particulgo: types of 'evaluation *"(e*g« , needs assessment survey techniques) 

independent of a^yi particular educational content or^level, and so are - 

^/-^- C 

Classified as of ''^general applicabii^ity" on' the- education diineAs ion ♦ Sate 
products focus on evaluation methods or skills (e.g*, data analysis techniques) 
iiidependent of any specific type of^^eval^^ and^so ^ classified as 
."general methodology" on the evaluation dimension v/ith potential applicability 
to any of the evaluation types. , ^ ' 

It should be remernbered .that the enphasis in the analyses of the evaliaation 
' training materials for the technical , paper to NIE (as distinguished from the 

Ctonsumer Guide to tracing materials for^potential users) is not on describing 
eacn product (in t^nos^ its placenent on the conceptual map>, but rather 'is 
^ on sutmarizing the densiiy and ovei^lap of coverage of various areas on the* 

4 

conceptual map by existing evaluation training materials in order to facilitate 
tne* making of judgrtHnts regarding desiraHTe areas for future develcpnent of 
such products. It was not always easy to classrfy the training materials 
* according to their^ evaluation and education en^)hases. Although fwo raters 
^ . ^ -achieved substantial agreement, other raters mTght disagree, with soitb of the 
classifications, ot Jjidividual products. Ihis classification system, ±h^, is 
most lilcel^ to be vAlid and~>^usefiiL when used for .its major intended purpose — 
. indication of the extent of coverage of evafuation and education areas by 
existing evaluation training materials — rather than for describing 
individiaal products. In Edition, the conceptual m^ should provide reackrs 
with a blueprint for analy:^g a wide range of materials ^and products (not 
.neoesfearily just those produced by RSD laboratories and oent^s) with regarji 
to the density and overlap of their ddverage. - " ~ 
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■ Classificat ion of Evaluation Training I^laterials 

• ^ Tables 1 and 2 ^sen, . ^ „^ ^ ^^^ 

P^ducts analyzed that ^ cl^ssiaea ^ ^ ^ 

eduction d«ain di:na„sio„s of the o^nceptual ^. ^^ astailed analyses 
. ^-i^ Individual oeu fre^oies aa» by laborato^ ;rcducers , are / 

.incl^. in the appendix (Tables la, 2a, »d 3a,'., Mso ^ the apf«,di. ' 
(Tables 4a aM 5a) a^ classiacation, of i^lvidual products identified by' 
prod^er and catalogue n^ter (aa sj^cified in the Censor Oa^, . 

It be seen in Table 1 (classification of trai.i.g-„.terial. on 'the' ' 
-i-tion di^ion of the oon^ ^, evaluation, training 

-terials de^.loted by ^ laboratories ^d centers, focus 'on the 'collection 
and.analysis of . data aspects of evaluatl™^ Cnly »out half the products 
deal with fooKir^ the_ evaluation (i..e. , 'deter^^^ its purposes or the 
q-stio. -ifis intended- ,toa^,,.,^^^^.^.^^^.^^^^ ' , 
.the b^is Of oollectad data, or «^caUng/reporting evaluaUc^ 
, -f-^ayon to others. ^ ^^t e^c^, ^ ,,,, ^^^^.^^ , ' ; 

evaluation ^ ^ traditionally considered to be as n,u* a part of 

evaluation, involving ™nag»^, <22 ^r«nt, , interfers^^ ,33 
or analysis, and consideration ^, c^straints on the evaluation (6 .™, are 
a^resi^d by considerably f^ pr«iucts.' (As c» be seen in Tableaa in the" 
a«^, this infr.^, ^ ^^^^ ^^^^^^^ ' 



evaluation. Materials »ere StT^^t^t^^ Purposes or cfuestipns for the 

they just indickted Se^SeS pj^e 2 wj"^"'"? ^ ^ ^^^^ " 
tiver.fomative, or s«JS^rt?"^r^^ °* evaluation (I.e. ,• prefor.a- 
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^ABLE 3 . Number and Percent of Products for Each Category of the Evaluation 
Dimension of the Conceptual Map' 



9 ^ 

* (Total number of pzD^uc±s classified = 36) 





^ ASPECT OR TifPE ' 
QF^IVALUATION 

r 


PR3DUCrS ADDI^SSnvJG 
ASPECT 


PRDD(/CTS SPEaEYING 
~ HCW TO PERFORM ASPECT, • 


/ 


Total Nuntier 


Percent 


Total Nuifiber 


Percent 




Prefonnative ' 


15 


42% 


11 


31% 




fbiinative*^ . 


. 18 


50% 


6 . 


17? 




SiMnative * 




1 73: 




6% 




General J^thodology 
wit±iout specific 
evaluation type 




19% 


6 


17% 




Collecting Data * 


33 


92% 


19 . 


T 

53% 




Analyzing Data - 


26 


,72% . 


13 


36% 




J — 

s^^laking Dec^iengor 


17 


■ 4.7% 


7 


19% 




Reporting Infonnation/ 
Judgments 


16. 


44% 


8 - . 


22% 




Interface with Users/ 
Sources of Infoaanation • 
of Data i 


9 


' 25% . 
f 


. ' 4 ' 


11% . 

' y 




Rx:aising Evaluation 


20 


56% ^ 


, 13 ' 


36% 




Adnunistering Evaluatioif. 




2^% , 


4 

i- 


11% 




Detemining Constraints 
on Evaluation 


2 


6% . 


' . 0 


0% 
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(e*^g. how c3o people in the 6oimrunity react to th^ program? doe^' the program 
provicJe instruction or training in .areas < for which there are jcb markets or"^ 
national n^ds?) . It is interesting to not^ that irost o^ the training produces 
analyzed do not specify a level of focus" at which the education^ situation 
is to be evaluated and do not provide training in how to select the level (s) 
or hew to ODnduct evaluation specific to, the selected l^vel (s) . Jtost pf the 
prmucts that do speci:fy a level focus, on the instructional level; hcaie of the 
iuGts focus on the societal level. . ' * 

The summaries of the training materials' coverage^ of evaluation and ^ 

education aspects (Tables 1 and 2) suggest that there are relatively few 
products addressing: sunmative evaluation atid the nontechnical aspect^ of 
evaluation (especially with specific hov-to training) , evaluation concerning 
societal- level and learner-level aspects of education, or evaluation of the — 
oomnunity and family aspects of education. This information " should be useful 
to ^those who need to inake decisions about directions for future develo^OTent of 
evaluation training materials; however, judgments' about the relative irrpprtan^ 
of the various categories on the evaluation and education dimensions of the 
ccaioeptual map and about the need for training materials addressing^ those 
^catiegoAes would have to be made before such decisions about development of — 
mterials oould.be made. It must be remembered that this analysis of U^ainir^ 
products has been relative to a conceptual map representing theoretical 
intersections of aspects of educational evaluation. The conceptual map was 
refined on the basis pf reviewers' conments sd as to#represent aspects o£ 
educational evaluation and their interrelationships as COTprehensiyely and 
parsimoniously as possible, but t^e reviewers made no judgments about the 
relative inportanoe for evaluators of categories or^4iieir^*i^ersectioni5.^ Sane 




sush judgments would nave to intervene between the sunmary data from ouf 

analysis and decisions about future product developfnent, G^s in coverage of 

certain areas revealed by .our analysis do not necessarily" inandate the 

develc^pcrent of training inaterials to address those areas. In addition, our , ^ 

analysis indicates general trends in overlap and gaps in coverage, but it does 

not indicate whether products that address the sane aspects of evaluation are 

. . ■* ^ * 

Redundant or ocffrpleiteabary,, or whether tiiey adequately cover (singly or dn 

cx3iTbination^" the aspect addressed, A detailed, content analysis of materials 

would be necessary to determine, for exaitple , whetl jer there is a need fpr more' ; 

training irateirials on collecting information even though *92 percent -of "the" 

products analyzed cover that ^gispect of e^valuation. ^ ' • > 

In the next> section of this paper, ve will present a detailed exanyle^f 

how an educational situation requiring evaluation be described in terms of 

the evaluation and educaition aspects (of the conceptual map) it involves, and 

of how the classification of evaluation training materials in terms of the 

r 

conceptual map cai^ be used to construct possib^le configurations of training 
materials to meet the evaluation needs of that educational situation. In order"* 
to make more extensive reconmendations 'for developiffent of tftraining products or 
Df configurations of use .of existing products; detailed content analysis and . 
field testing of materials and configurations of materials ^^uld have to be 
conducted. - . • - ' ^ 
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) III . An Example of an Educational Situatiorf Described in 
Teritis of Conf iquratiols of Evaluation and Education Aspe cts 

>in .order to present a detailed application of the conceptual map, let 
^us si^^pose-that an educational agency is interested in evaluating an innovation 
linkage program they are establishing. This program is intended to provide a . 
variety of support services to schools interested in adopting educat^i6nal 
innovations. The agency^ is basically interest^ xn^the following elanents of 
the program in terms of evaltetion: 1) the background and needs of the ^ 
schools seeking services f rem the program, and selection of a subset of these 
schools for further stufy; 2) the types of services the ptograra offers as 
well as v^en and hew it delivers them; 3) the decision-making process that a 
school undergoes in order to select an appropifiate educational innovation; ' 
4y^e inplementation process^ that a school undertakes to adc^t a selected 
innovation; and 5) the outccme or impact that the innovationyh^^^for the 
appropriate population {e.g^ students-,, teachers, teacher aides, e,tc.). 

Before we begin our analysis oft these various elonents in terms of the 
ccaiceptual map, we should like to point out that in our initial discussion 
of the evaluation process dimension (Figure 1) of the conceptual map, we 
described the activities of focusing , administering , interface , and caisiderirg 
institutional oy system constraints preceding or overlappirg the other 
evaluation activities. In the analysis of the elements below, we assuire that 
these activities are embedded throughout the evaluation process we describe, 
ilqveyer, for each element, we will p6int out those evaluation activities we ^ 
feel would be of particular concern for that elonent. 

< • 4 

Elennent^l, Schools' Backgrounds and Needs 

Element 1 of the evaluation activities described above would involve 
,a pireformative (A) type of evaluation. The specific evaluation activities 
enccnpapsed by this element include: collection of data (III) regarding the 
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eciicational prdDlems and needs of the schools in\70lved in the program, their 
history of innovativeness and the effects of tljose innovations; the analysis 
and interpretation of the data (IV) to determine the propensity of the school , 
to innovate as well as characteristics of the school that may facilitate or 
inhibit iirpleinentation and impact of the innovation; and the rendering of 
judgments or decisions (V) regarding a subset of schools \>ath a high or la^ 
degree o£ success at innovaticn inplementaticn and^use^ This element is also 
concerned with the selectiai of a subsairple of schools for intensive study * 
whida would involve focusing (I) and administering (M) evaluation activitie^* 
Constraints (CS) in the evaluation at the various sites v/ould also be a factor 
^/hich would probably have to be consider^ in selecting the sites for further 
evaluation ♦ . 

Using the educational dormn dimension of the conceptual map (Figure 2) , 
we can identify the educational context within v^ch Element 1 will be evaluated 
as consisting of school settings (A) at the, institutional level . (2) focusing 
on policy (a) , resources (b) / programs/curricula '(c) , and products (d) contents 

The. configuration of Element 1 on the conceptual map, then, could be 
labeled as "A, I, III, IV, V, AM, CS" oi the evaluatiai process dim^sion and 
"A, 2, a, b, c, d" on the domain of education dimensiorfv^ 

Elen^nt 2^' Support Services ♦ . , 

The evaluation of this elanent woilld be of the formative (B) type that * ^ 

occurs during the activity cycle of the program and provides data that feeds 

• * 
into the > program as a basis for inproving program design and delivery ♦ -The ^ 

specific evaluation prooess activities would include for this elor^ent: 

collection of dat^ (III) regarding the types of services 'provided by the 

linkage program during eadi school'^ decisioi-making and irrplementation phases 

of innovation adoption; the analysis and '^interpretation of the data (IV) 



to provide feedback to the linkage program for inprovemant of services and 

program operations; and reporting tlie results (VI) to the ^linkage program to 

provide, a basis for modifying, refining, and inproving' program services. 

The e^luation of this particular elorent is primarily directed toward 

•kf * • * 

internal itonitoring of the linkage program's activities. Accofdibgly, 

administering (AM) should be included as part of the coif iguration of evaluation 

'CCn^xjnents. Constraints (CS) would also be involved -since in this case the 

evaluator is also part of the staff of the educational agency which ''is, develpping 

the innovation linkage system being e\>aluated. Interface (IF) con^J^s^fps v/ould 

t * . ' , * 

be iiTportant as the, specific needs and characteristics of the 'people to ^/^lon 

the information is to be reoorted must 'be considered. ' ' s 

The educational -context in which Elorent 2 would be evaluated can be 

defined by using the education dorain diJT>ension of ,the conceptual map. 

Essent^-ally, this context would consist of professional organization settings (B) 

at the institutional and instructional level (2,3) focusing On resources (b) , 

^ — 

programs/curricula (c) / products (d) , and people (e) content. 

The configuration of Elenient 2 on the dimensions of evaluation bxA 

education can, then^ be classified as "B, III, IV, ,VI, AM, CS)' IF" -on the evaluation 

process dimension ahd "B, 3, b,^ c, d, e** ai the education domain diironsion.' ' 

« 

Element 3. Decision-Making Pro^^s c 

The evaluation of this element is primarily sumnrative (C) , though it 
ooiold also have some aspects of formative (B) evaluation. Essentially, the-^ 
evaluation would be concerned with the description of program related phenonana 
that have occtired so that future analysis and' interpretation can be conducted 
to 'determine the utility of such phenofnena. However, it is condLivable that . 
the data wuld be useful in helping the linkage program better integrate its 



input and services with the specific decision-making functions tliat a school^ 
progresses throu^; thus, the evaluation could have a fomative cast to it. 
The specif ic* evaluation process activity involved in the evaluation of this 
elenient would consist of the coilecticn of data , (III) rec?arding eadi 
schooljs method of refining and verifying the educational problem they <ire 
addre^ing, its process of reviewing innovations, and what and vAiy a particular 
innovation was selected. If feedback is to be given to' the linkage program, 
the ev^luiation activities woxiLd al^o 'include analysis and interprctaticn 
of tile data (IV) , and reporting the results (VI) to the linkage agent. 

The thrust of the evaluation .of this elansnt is direct^ toward the 
schools as well as the internal program monitoring involved in the previous 0 
element and will involve focusing .(I) and administering (AM) the evaluation, 
^ince data will be collected fron people in the schools,' interface (IF) concerns 
will be inportant. The constraint^ (CS) w^re previously considered for the 
evaluation of Element 1 and probaBly do not have to be addressed again for this 
elonent. ' * , * > v * ^ 

The educatioial context in v^ich element 3 would ^ be evaluated consists of 'the/ 

\ ... * . ' 

.school settiii^ (A) and the institutional level (2) focusing on a policy content (a) 

* i . • . ^ : • ' f * . ^ 

Overall, then, the oonj|iguration* Element 3 can be classified as ^ 

"I, C, III, IV, VI, AM, IF" tn the -evaluation process dijnension of tlie conceptual ^ 

map and "A, 2> a" on the* education donain dimension of the map. 

Elgneht 4. Inplementaftion Process 

The evaluation of Element 4' is similar to that of Element 3* in that * ' 
it could involve both formative (B) and sumiative (C) types of "evaluation. 
Feedback to the linkage' program could enable it to inprove its services to a 
school during the iitpienBntation stage and the data could also be useql bo 
determine the effectiveness of a school's irrplemehtation strategies! Ihe 
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evalijatian process activities" include, as with Elenent 3: the cyllecticai 
. of data (III) regarding the process a school undergoes to install, an 

r 

educational innovation, vMch nay include activities such as^ staff and connunity 
-orientation (preparation), and, if applicable, acquisition and distribution 
of naterials-, staff training, program administration, ^etc; analysis and 
interpretation of the data. (lyi , and reporting the results <VI) to the linkage 
progran for purposes of program modificatipn and'iitprovement, 

, Internal program activities and school iirpleinentation acti>KtieG will be 
evaluated, thus the evaluation of this elorent is similar to the 4valiaation 
of the preceding elenient and the oorponents of focusing , administering , and 
considering constraints need not be specifically addressed.*^ Note once again, 
however, that th^be three corponents permeate the entire evaluation process 
•and must be continually considered at a general level. Since data will be 
reJXDtted to the -linkage ^program, interface (IF) concerns will be iirportant for 
this element. . - ^ ^ " , • 

Bie ^educational context within vMd^ the evaliaation of Element 4 will 
be oOTducted can be identified on the education domain dimension of the 
conceptual map -as consisting of' the school settings CA) at the institutional 
level C2) and focusing on policy Ca) and people (e) content s ^ . ' * 

^ The overall cpnf iguration of Element 4 on ^the evaluation aiid education 

. . • ^ ^ ---- ^ / ; 

"dimensions can, then, £)e classified as "B, C,^III, B7, VI, IF" on the evaluation 
process dimension and *'A, 2," a, e" on the education donciin dimension. 

Element 5. Innovation Outocmes 

. The evaluation of this element in many respects subsumes the previous 

elements. , It is a sxitnative (C) type and can use the data from the previous 

<) 

elements as a part of its process activities. These activities include: 
4 ' • 

the oc511efct3.on of data (III) regarding the irtpact of the innovatiai in temvs 
' - ' ■ ■ 36 41 



^ of stuient pea^fontance in the prdDlon areA, teacher behavior, and connunity 
involvement; analysis and interpretation of this data (IV) in terms of dWQes 

,..„in the scope and intensity of the educational prdblon being addressed in each 
school, and changes in teacher behavior, student performance, and comnunity 
involvement; judgments and decisiois (V) regarc^ULng the t^/pes of schools that 
most effectively used the sijpport services of the linljage program the utility 
of support se5svioe delivery by the linkage system, the relationship of impl^nen- 
t^ation methods and iiiplementation outconas, and the contribution of the linkage 
progr^ to the inplo^nentaticn process *at each of the schools; and the reporting 
and dissemination of the results (VI) of the evaluation to policy makers with 
recamendations regai^ding funding of similar programs and alternative program 
designs. ^ ' . ^ ^ 

e 

This particular element is virtually an extension of the previous elCTients 
with sane unique facets of its own that contribute to the evaluation problem 
(^•g*, collecting information on teacher behavior and student outcomes).' 
evalualu.on process ccnponents of focusing and administijfition should entail 
much the same considerations as were inyolved in Elonents 3 and 4. The 
coT^nent tx^nstraints (CS) , hcw/ever, may have to be considered anew, as the 
release of the final report has -iiiplications for educational policy and funding 
v^ich may have political undertones. Interface (IF) concerns are of 
particular importance for this' eloient as data will be collected from and 
reported to a variety of people. 

In teims of the education domain dimension^, the evaluation of /Elorient 5 

will occur in the educational cqitext of schools (A) r professiaial organizations (B) 

and ccninunity settings (C) at the institutional^ (2), instructional (3), and 

learner (4) -levels, focusing on policy (a), resources (b) , programs/curripula (c) , 

■>» 

prodQcts (d) , and people (e) contents . 
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The evaluation of Elem^t 5 has a general configuration on the dimensions 
of evaluation and edgcaticn that can be' labeled as "C,/III, IV, V, VI, IF, CS" 
for the evaluaficn process dijnension and ''A, B, C, 2, 3^, 4, a, br c, d, e" on 



the educaticrt donain' dimension. 
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IV, Ex;an)pres of Configurations of Evaluation Training 
^terials to Fit Configurations of Evaluation Functions 

Now that we have deniohstrated how an educational evaluation situation 
may bfe described in terms of configurations of job functions or activities 
{eval'uation processes within educatiolial contexts which are delineated by ^ 
oocqponehts of. the conceptual map) , we will illustrate briefly hav configurations 
of evaluation training naterials ^propriate to the job function configurations 
can be identified. Since an evaluation training program is unlikely to have 
such a specific job function focus, we will then .present soms exanple 
. configxarations of materials that might be appropriate for the more likely 
program v^ch would be designed to provide training in a much broader range 
of educational evaluation functions and ^activities. 

A* Configurations pf Materials to Fit a Specific Evaluation Function 
Conficnoration 

We would not expect to ^ind evaluation training materials specifically 
tailored to all aspects of any particular evaluation job/ In selecting 
Qonfiguraticns of training materials most appropriate for the configuration 
of evaluation and education aspects of a job function, it is 'iinportant to 
decide v^at aspects of the job function most need direct relevant training. 
Taking one of the simpler elements (jElaftferit 1,T of the- evaluation of the 
innovation linkage program described above as an exanple of a specific evalua- 
tiai fijnptiaiV:onf iguration* for vs^dch we want^to find relevant training^-* 
materials, we find4±iat there are several evaluation training mat^erials 
available that a3dress sane of the aspects of ^evaluation involved in this 
element, but none that aidress all of them. There are many possible configura- 
tions 6f relevant training npterials, depending upon what aspects of the jdD 
function' axe cpnsidered tp most directly need^ai$)licable training! Figure 4 
presents k sumnary of the possible configurations described below. • 



Figure 4. Examples of Configurations of Ti-aining Materials 
i^ropriate for Innovation Linkage Program, 
Elorent 1 Evaluation Needs 4 

' f 

Training material X ' 

Caifigioration (cxx3ed to Consuiner's Guide) Aspect (s) of evaluation addressed^ 



b 

i 


CSE 9 

* 


conducting needs assessment" at institu- 
tional level in school letting 
» 




RBS 4 


focusing, administering, and ' conducting 
needs asse^psment at learner level in 
school setting 


II) 


CSE 2 

* 


orientaticai to needs assessment in 
school setting > 


< • 


CSE 3 . 


• .* _ 
administering and conducting needs 
assessment in school setting 


> 


CEPM 5' 

< 


analysis and interpretation of data 
relevant to needs assessment at 
institutional level in scf\ool setting 


III) 


RBS 5 


administering and conducting evaluation 
in school setting 


f * 


CSE 3" 
' or . 


administering and conducting needs 
assessment in school seating 




CSE 9 


|bcnducting needs assessment at institu- 
/ tional level in school setting 



IV) - FV7L 13 • \ caiductinq evaluation 



FV7L 14 \i:^strunBntatioo for evaluation and ^ * 

^sQnducting evaluation 

FWL 15 • " administering and conducting evaluation 

CEPM 3 conduct inc ^valuat icai in school setting 



CSE ^ * cpnduct in( 

or 



i 



neqds assessment at instSEfr - 
tional/ level in schcx^l setting * 

PBS .4 focusing , administering , -and conducting 

needs assessmerft at learner level in 



school setting 
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If, for exaitple, we ^'decide that it is necessary to have training in the 

evaliiaticn cxxioepts and procedures -specific to pre formative (A) evaluation at 

the institutional (2) level within a school (A) setting, .the CSE Needs 

AssessiTBnt Kit^ (CSE 9 in the qpnsunier's Guide) seei|g most appropriate, for 

there is a reasonably close match between the conf igioration of evaluation and 

education aspects of the job function ("I, A, III, IV, V,^AM, CS/A, 2, a, b, c") * 

and that of the training material ("A, III, IV, V, VI/A,^2, 3, a, b, c''). Both 

the training n^eds kid materials involve collecticn 'and analysis of data about 

school policy, resaifce and program needs ancj decisicn-making about the 

adequacy with which these needs are being met. However, CSE 9 would not bQ 

ccxT^letqly adequate by itself because it does not include training in focusing (I) , 

administering (AM) or analyzing constraints (CSh on the evaluation • The RBS 

l^upil-Perceived Needs Assessment Package (RBS 4) might be a useful conplement 

to CSE-9 as it does* address focusing, administering anJ ar)aly:^is of constraints 

in preformative evaluation ccnducted in a school setting, though (as indicated 

by its title and its* description in the Consumer's Guide) it limits its focus 

— ^, — — , 

to pipil perceptions of needs . As can be seen in I'able 4a, (Afpendix C) ^ 

there are several training materials that address aspects of preformative 

evaluation and could provide training 'relevant to the preformative evaluation 

functicns needed in Element 1 of the innovaticn linkage program evaluation. 

Fox example, cne might conbine the CSE^ QrientatiOT and Needs Assessnent 

Vforkshops iCSE 2, and CSE 3 respectively) td provide an overview of evaluation 

(including needs assessment) as well as training in specifip techniques useji 

» 

in nee^ assessment* Combining these two workshops, the "trainee WDuld receive^' 

(/■ - ■ ^ • 

training, relevant to focusing (I) and administering (AM) evaluations ^s well ^ 
as to 'the collection and interpretation of -data. However, though these 
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'v^rkshops deal specifically with preformative evaluation, neither of them 

s 

focuses on evaluation at the institutionaj level, which is ths major thrust 
of this element of the innovation .program ^evaluation* The Assessing Achieve- 
ment of the District's Broaa Goals packetT (CEPM 5) , which does focus on 
institutional (and instructional) level evaluation, could ^be a very useful 
supplenent to the CSE workshqps, ,CEPM 5 by itself would -probably not be 
sufficient for the needs of Element 1 , since if does not cover the actual • * 
techniques of needs assessment in aS moch detail as the CSE workshops. 
Evaluating a Curriculum Program (RBS 5) , which addresses general planning, 
inplemsnting and administering evaluations in a school setting with training 
materials that deal mJore specifically with^preformative eveJ_uatian techniques 
in school settings, such as CSE 3 or CSE 9, > ^ » 

-^Anpther ^proach would be to lose materials that provide detailed training 
techniques of evaluation planning and implementation but do riot eitphasize the 
special problems or needg of evaluation done» in school settings or by school 
personnel. A serie^ of R€i modules (e.gljFWL 13: Measurement and Testing , ' 
PMi 14: Design of Evaluation ^Instruments and ML 15: ^ Planning and Inplar^ent- 
ing EvaluatJtor^) could be useful in this regard, but prob^ly should be 
combined with materials that .focus on school settings, such as CEPM 3: 
Instructional Program Planning, Evaluation and Camrunication and perhaps with 

materials that focus on fieed^ assessment, such as CSE 4 or RBS 4. ^ 

< 

It qan be seen fran this discussion of possible configurations of 
training materials ^prcpriate for the configuration of evaluation require- 
ments of a specific evaluation jcto function (Element 1 ot the innovation 

-* 

linkage, program evaluatiai) that there are many possible materials configura- 

t ^ 

ticHis and that selectiai of 'a desirable configuration depends upon the 



particular aspects of the^job functicais that are felt to"need directly and 

specifically relevant training. It, is unlikely that any trainirt^ material or. 

configuration of materials will fit highly specific evaluation traini^ needs 

precisely;^ even a well-chosen configuration of materials may have to be. 

sipplemented by instructicn and applicatioi tailored to those specific needs. 

is iirportant to be able 'to select ccnf igura£icns of training materials 

that are as ^propriate-as possible. It should be remembered, then, that 

though the classification of training materials in terms of the conceptual 

map (the class if icatiai presented in Tables 4a-6a in Appendix C) should be 

useful in identifying con^iguraticns of materials likely to be appropriate for 

configurations of evaluation functions, cont^^ analysis (or at least 

consideration of descriptions) of individual materials would also be djiportant. 

*■> . ^ , 

B. OonfigurationV of Materials to Fit Broad Evaluation Training Needs ^ 
Since few evaluation training prcpgrams are likely to haye siich^a specific 
job function focus as described in ths preceding section, we will. now present 

6. 

some examples of possible caifiguratiais of training materials that might be 
ajpropriate for a program designed to provide draining in a much broader -range 
of educational evaluation functions and activities. The classification of 
materials in terms of the cdnceptual map would probably be irore useful in ^ 
selecting configurations for more narrov and specific needs, since the 
particular goritent foaas of the materials would not be so important. 

For exanple/ if a progirraifn were concentrating on formative evaluation, one 
could select a caif iguration of materials which would provide a balance of 
1) detailed procedures and methods (e.g. instrurrent development, data analysis) 
of evaluaticai in general, 2) specific problans, concerns and methods of 
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fontative evaluation and 3) specific problems, concerns and methods of 
different types of formative evaluation (e.g. product evaluation, program 
evaluation f evaluatiai in school settings) . There would be, many possible 
configurations of naterials which -would present such a balance of training. 
Figure 5 presents some of these possibilities • 



Figure 5. Examples of Configurations of Training Materials 
T^rcpriate for Training Program with Formative 
Evaluation Focus 



ODnfiguraticn 


Training, material 
(cxxded to Consumer's Guide) 


Aspect (s) of evaluatioi^ addressed 


I) 


FWL 8 


introduction to - types ,and 
niirno<?es of evaluation 




FWL 10 


data analysis and collection 
methods — 




FWL 11 


aetaiieo purposes ana proceaures 
df formative ^valuation . 




' SEDL 1. 


managing and conducting, product 
fomrative evaluation 


II) 


CSE 2 


orientation to types and purposes 
of evaluation 




FWL 13 


measurement and testing .met^iods 




CSE 5 

» 


detailed purposes and procedures 
of formative (inpler^ntation) 
*• evaluation 




' FBS 2 


manaqinq' and conducting program 
format iv§ evaluation " ^ 


^^III) 
• 


XEPM 3 
SWRL 1 ^ 


introductie»i to types, piorposes," 
3nd methods of evalu^ion 

research and evalu|.tion methods 




CSE^6 


detailed purposes and procedures 
of formative • (progress) 
evaluation 




SWRL 2. 


"managing and conducting product 
formative evaluation 



similar configurations could be suggested fqr other * training program 

evaluation thrvtets, such as specification and evaluation of learning outcomes * 

reporting evaluation results, maintaining snKXDth relations with people involved 

in* the evaluation process (i.e. interface), planning and construction of 

evaluation instruments, etc. In all these cases, configiora tions of training 

materials could be selected by reference to the classif ideation of these 

^ i . * * 

materials on the conceptual map of educational evaluation. Hcwever, several 

admonitions should be macje regarding prescription of training 'materials based 

solely on the classification of those materials relevant to the conceptual 

First , in order* to determine the feasibility of using particular 

materials as well as their ccmpatibility and conplanentarity with each other, 

one would need t6 knw about several pragmatic aspects of the materials not 

included on the conceptual map, siich as cost, time and^sourcd requirements * 

and intended users. Thi§ infoapatiort could be ^ound^y looking at tlie 

product description^^ in the (^sumer's Guide * . » " ■ 

In addition, it wctfld be^important to read the Consumer's Guide descrip^ 
' , " ^ . ^ r ' • 

tions or to look at the nSt^erials themselves in or^er to find out more about 
the specific content. Tne.gbais and objective^, instructional format and 
learning activities and user* prereqmsites would all be important aspects of 
"a training product to consider in selecting produci! configurations. Information 
about these aspfects is, in most cases, available in the Consumer's Guide ; 
hcwever, it would probably be "very .useful for a program planner trying^ to 
establish coifiguraticns of materials to supplement use^Df the conceptual map 
and the Consumer's Guide %ith a detailed content analysis to determine 
a material's content and approach. " . ■ • 

/ ^ , 

45 * * ' 
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It would be, of course, very important to know sortething about tHe ' * / 
quality of itBjierials before selecting configurations. In the Consumer ^s Guide , 
statanents were made as to how products could be best used, but'^^judgments of 
quality or adequacy were carefully avoided or downplayed. So, any judgments 
of quality would have to be made or arranged by program planners themselves . 
Several possible criteria and methods to use in making these judgments are 
suggested belcw: . . ■ ' ; 

• judged accuracy of field testing procedures reported in the Oonsuner 's 
Guide; 

• resjalts of field testing reported in the Consumer's Guide or obtainable 
frc«n product developers; 

• internal ^consistency of goals, objectives, content and learning , 
activities as determined by content analysis;. 

• appropriateness for target audience: clarity, level, degree of 
conceptual or practical afplication eipg^hasis ;. * ' ' * „ ^ 

• dep£h and breadth of coverage; 

• validity of content as determined by evaluation e5<pert; 

• practicality of content as determined by experienced ev^uator; 
adaptability of materials to partiaiLar program and individual needs. 

Anyone selecting ccnf igiorations of training mate^jicLLs .for evaluatiai training, 
progiams would probably need to consider* these various aspects of the 
materials that are not reflected in the classification of materials on the 
conceptual map. However, as we have shown, that classification can be very 
useful in at least preliminary selection. 
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V. CJonclusion 



The cxjncepUial franework of educational evaluation presented in this" 
paper has helped us d$velcp sane* of the categories of description we used in 
the Consumer's Guide to Evaiuation Training Materials and seens useful 
for classifying and analyzing evaluatiai job functions and evaluatioi) training 
naterials. Classification of existing evaluation training rraterials produced 
by educational laboratories or R&D centers in terms of this framework hassled 
us to tentative conclusions about areas or aspects of' educational evaluation 
that are or are not covered by these materials. These conclusions should 
provide sane useful input for those v*io will make decisions about future . 
develc^xnent of evaluaticai^ training materials • We have also shown hdv 
classif ication* of training needs and materials with this conceptuaL f rame\^ork 
might help soneone suggest possible configurations of training materials 
appropriate for caifiguratiais of training n^eds, EcMever, in order ,to more 
adequately determine desirable futiore decisions for product develc^ment or 
appropriate configurations of existing materials, additional -analyses would 
have to be undertaken. For either type of decision about training materials, 
analyses of ^product content and quality (e.g. detailed content analyses, 
analysis of product? in operatiai) and analysis of a broader range of available 
products (not limited to those produced by educationai laboratories and R&D 
centers) wGuld have to be conducted. In addition, to determine directions ♦ 
for future developiient, judgments would have to be nade about the relative 
importance of the Vcurious areas/ and aspects of educational evaluation 
delineated in the conceptual fraitewprk. 
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APPENDIX A , ' 




CS'E EVAlUAflON WORKSHOP I: AN ORIENTATION (Evaluation Technologies P-rogram), 
INTENDED USERS 



o 

CO 



ro 



Educational personnel including: superintendents, evaluation specialists, 
project directojts, and other administrators; teachers; graduate sjtudents; 
researc^iers and curriculum and program developers; and project monitors. 
Individuals .involved in- evaluation in social action programs , such as health 
and social sciences, have also found participation beneficial, 

♦ 

TYPE OF EVALUATION ADDRESSED 

The workshop treats activities and procedures appropriate to different phases 
of evaluation: .needs, assessment, program planning, fomiattve evaluation, and 
sumnative evaluation. ^ * ^ 

APPROACH TO EVALUATION • . , 

The workshop expresses the viewpoint that the purpose of evaluation is to 
provide information that will facilitate and improve educational decision- 
making at all levels. Evaluat>on questions are related to four major phases 
of program's cycle: needs assessment, program planning, fqrmative evaluation, 
summative evaluation. ^ ^ / 

MAIN EMPHASIS - . , ' 

The workshop focuses on major steps and components in the evaluation of 
educational programs • 

GOALS AND OBJECTIVES ^ ' . 

Major workshop goals are to enable participants to understand what kinds of , - 
information an evaluation can provide 'for educational decision-making; and to 
understand the general procedures and. problems involved in selectinj^ colle9ting, 
analyzing, and reporting that information. < 



Participants become familiar with what ts-^^involved in: conducting a needs- 
assessment, building the evaluation design into a program pTari,' determining 
whether a program is being implemented properly, assessing its progress, and 
presenting evaluation data to different audiences, • ^ 

MAIN ACTIVITIES • * 

Participants are first given instruction in a. phase of the evaluation process, 
then work with other team members to solve a problem common in that phase'. 
After this .exercise, the workshop -leader provides concrete feedback on how this 
problem could have ^eep 4vandled and conducts a discussion of alternative " 
solutions, . ' - * ' 
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PROVISIONS FOR USE • • 

A. Format: Workshop , 

B. . Personnel Needed : Requires a leader familiar with workshop material either as 

a result of professional evaluation experience or previous parti cipati.on 
In a workshop session. Upon user request, CSE can supply a workshop leader. 

One qualified leader for every 30 -jifr-pai^tici pants is required . ^ An 
assistalit familiar with the wc5rkshop is recommended if there are more 
than 30 participants. • * - , 

C. Product Components ; A Leader's ManuaT describing the step-by-step procedr 
ures for organizing- and conducting the^workshopi a Participant's Notebook 
for each learner containing instructi&n, reference, and reading materials; ' 
one set of exercises for each three participants; and pre- and post-tests • 
for each participant. , • * - 

D. ResoQrces : A blackboard and other classroom equipment is recommended; 
refreshments and lupch served close to works^hop location assists ttje 

- leader in keeping with a tight -schedule. , ^ 

E. Time Needed^ Two full days of instruction (^approximately 14 hours). It may 
be acj^pted a series af 4-hour sessions. / I x 

F. Conditions of Use ; To conduct a workshop, an individual should have 
participated in a prev.ious .session or have sufficient evaluation experience 
and familiarity with workshop content. ^ ' . 

In most sitiiations, a minimum of 21 individuals should participate; It is 
desirable that the entire group of participahts participate in the* entire 
ession sinbe teams of three work on exercises together throughout. ' 

APTABILITY r . . i 

The workshop can> be used in a variety of organisational settings- 7 .Groups of 
individuals. from' a number of different organiza^tions anti levels of experience 
can participate nn one session; an organization can^use the materials ;to train 
its entire staff^; a professor can use the materials 'in a course for graduate f 
stiidents.* The.wprkshop may also be ,incorporate,dJn a longer evaluatiqn \ 
training sessi6n^:run by CSE wh*ich includes instruction tailored to a particular 
group's needs 4nd interests. This workshop can; be used separately or in conjunc- 
tioh^ wi.th other progranimteria.ls. ^ ^ , 




RELATED PRODUCTS \ 

Other. CSE iworksKOps ; and FWL, Evaluation for Program Improvement. 



COST ' ^ 



Range of cost ii as stcited on page 2. A full set of materials for the workshop 
consists of: i ' ^ ^ . ^ .( 



Participant's Notebooks (1 per participant, reusable reference); Exercise Pads 
(1 per team of 3); Pre- and Post-tests (1 pair per participant); Leader's 
Manual (1 per leader); Audio tape •(optiohal — 1^ per leader, included viith Manual, 
reusable) • - ' ♦ 

\ • / . 

AVAILABILITY , ^ ^ 

The worlcshop is availait^le from CSE Field Services- 

HISTORY AND EVALUATION 

Formulation of the workshop's content and organization was guided, by the CSE 
'view of the evaluation process. This view, in turn, was generated .by CSE's 
experience in conducting educational evaluations as well) as by the writings 
of recognized experts in the field and the suggestions of educational 
adninistrators and^ evaluatorsJ^ ^ 

The first draft of the workshop". went through a series of ' feasibi lity tests- to 
detennine its appropriateness for the target audiences and tq identify compo- 
nents needing modification. When the nepassary revisions were made, the work- 
shop was field tested by CSE staff to ensure that it achieved its objectives. ) 
At th^ conclusion :of the field testing, the workshop was again revised and ' j 
then .released for ' operational testing . The operational testing was conducted | 
by^non-CSE staff and provided a check^'bn whether the ^workshop still achieved \ 
-its objectives in.its final form and under the wide range of conditions in 
which it would eventually be used. A follow-up impact study was also conducted 
in order to be certain that the participants used the . workshop training and ^ 
materials on their jobs. These field tryouts involved over 10,000 participants , 
from numerous loc^l, state, and federal agencies across the country. \ 

Information obtained from the pre- and post- testing, ^the questionnaires, and 
the impact study, indicates tfiat the training is^valu^d by the participants,' 
that many of the participants use the materials and implement thp procedures 
after'the workshop, ^and that the individuals would recommend the workshop to ^ 
their colleague^ anci would themselves participate in. additional workshops ' ^ 
developed in the series. ^^ ] 

-I-rv adfiition to de|/eTopment and field testing of the participant'^ material?, | 
the -Leader's Manual for the workshop was thoroughly field tested. Once having \ 
participated in a^ workshop, 'an educational professional' can competently train | 
colleagues.' Dataj indicate that workshop leaders coniJuct tjie session in the ] 
manher prescribed! in the Manual. ' ^ '« 

COMMENT ' / ' • • . ^ . ' ^ , ^ ^ j 

Testing of the CSt, Evaluation Workshop I has been exjtremely thorough. It has ^ 
'included evaluation of the amount of use participants have made of the materials 
in their own settings, and of the ability, of inexperienced leaders to conduct ' 
successful workshops. The materials themselves are convenient, attractive, ^ 
'and clearly writt^en and organized. . 



r 



The problems and activities described in the case study are those that a single 
evaluator rather than an evaluation committee, would have to deal with. There 
is a stfong emphasis, on evaluation design. The materials present a great-dgal 
of detailed information. Developers assume that participants will retain their 
copies of the materials, and it would probably be necessary 'to have them 
ava.yab1e for reference to use the skills covered in the Workshop. The case 
study focuses on tradittonal versus modern conflicts rather than ethnic or 
.racial problems. The case study also features no females in positions of 
aifthority* 
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APPEM)IX B ' 

Irdex and List of Products 
Described 



1^ ■? 
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•Training Material 

in tlvis duid 

Center for Educational Policy Management (CEPM) 

1 .School Planning, Evaluation, and Comnunication System (SPECS) 

2 A SystejBS Analysis of a School District (SPECS I) 

3 Program Planning, Evaluation and Communication (SPECS III) 
4. Community Based Broad Goal Definition (SPECS IV) 

5 Assessing Achievement of the District's Broad Goals (SPECS V) 
6 Strategies of Organizational Change System ' | 
7 Techliology for Organizational Change Specialists 

Center fon the Study of Evaluation (CSE) ^ i 
1 The Evaluation Technologies Program |, 
1 CSE ;Ev^luatipn Workshop I: An Orientation | 
3^ CSE ^Evaluation Workshop II: Needs Assessmentf 
4> CSE EvaTuatiorv Workshop III: Program Planning ] 
. ^ CSE, Evaluation 1^orks^|Op IV: -Irnplementation Evaluation 
6, CSE: Evaluation Workshop V: ProgresLS Evaluatip 



ipr| 

7 CSE Evaluation Workshop VI: ['Outcome Evaluatior) 
8' CSE Elbmentary School Evalualion Kit: Needs Assessment 
Si CSE Needs Assessment Kit 

10 CSE Program Planning Kit ^ J ) 

11 CSE Prpgram Evaluation Kit ^ | 

I i ' . 'r 
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Far West Laboratory for Educational Research and Development (FWL) 

1 The Educational Management Program - - * 

2 Analyzing Problems (Unit 2) * * . 

3 Evaluation fpr Program Improvement (Unit 5) 

4 Development, Dissemination, and Evaluation ,(DD&E) Training Resources^ 
5 DD&E Series 1 : Planning* 

6 DD&E Module 1.1: Problem Definition and ^Specifi cation qf Outcomes 

7 DD&E Module 1.2: Consideration- of Alternatives 

8 DD&E Module 1.4: Introduction to Evaluation 

. 9 DO&E Series 2: Information/Data Collection and Qrganization 

10 DD&E Module ;2. 2: Data Management 

11 DD&E Module 4.4: Review^, Tryout,' and Revision 

J2 DD&E Series 5: Evaluation ^ ^^ . v " \ . • 

13 DD&E Ijjodule 5i^2: Measurement and'^esting fpr^ ppvelppers and 

' Evaluators 

14 DD&E Module 5.3: Design of Evaluation Instruments 

15 DD&E Module 5.4 Planning and Implementing Evaluation' 

ft 

16 DD&E Module|5.5: Evaluation Problems ' " - 



Northwest Regional Educational Laboratory^ l NWREL ) 
1 Objective Analysis and ManneJd Change Series 
2 Research Utilizing Problem Solving (RUPS) 

S ' 

3 Preparing Educational Training Consultants (PETC) series 
4 PETC: 3 - Organizational Development ' i 



GO 



< • 



Research for "Better Schools , .Inc . (RBS) 

^1 Inventory for Curriculum and Instructional Improvement 

2 Handbook of Comprehensive Planning in Schools * 

3 Surveying Your Community 

4 Pupil Perceived Needs Assessment Package 

5 Evaluating , a Curriculum Program _ 

.6 Curriculum and Instruction: R|^annincj/ Improvement 



SpMthwest Educational Development jaboratary (SEDL) 



1 CALIPERS 

2 A Developmental Process 



Southwest Regional Laboratory (SWRL) 

« 1 Instructional Pr^dduct Research 
2 Instructional Product Development ^ 
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APPENDIX C 
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TABLE la. Nuaber of Products by Agency (Lab or Center) for Type x Aspect 
Cells of Evaluation Dimension of Conceptual Map 

1 

(numbers in parentheses are for products which-/^ 
provide specific how-to-training) 



Aspect of Evaluation 



T^pe or Purpose 
of Evaluation 
(Laboratory 
produce) 


Collect 
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Judge/ 
^ Decide 


Report 


Total ^ of 
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Products 


Prefonnative 
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TABLE 2a. Nunber of Products by Agency for Cells of Evaluaticn Dimension 
of CQr)ceptual 

(nunbers'in parentheses are for products which provide 
specific hew- to training) 
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* Table ^a*, Specific Products Classified In TSppe x Aspect Cells Of Evaluation 
' ^ , ^intension Of (ionceptuai Map ^ ^ ^ . ^ / 

, X (* irxi^cates pipducts vMdti .provide specific how-to trainirfg) 







Aspect of Evaliiation 






• 1 




Type or Purpose 
of Evaluation^ 


■• Collect 


Analyze 


Judge/ ■ 
Decide 


Report 


« 


• 


Preforinative 


CEPM-2*4* 
CSE-2<3,9* 
FWIr-2*, 6, 


CEE^^S* 

CSE-2,3*9* 

FWL-*6,r- 


CEPM-5 
CSE-3*9* 
' •FWL-6,7 


CEPM-2 

CSE-2,9* 

FWL^7 




e * 


Y 


^ 7 
NVJREL-4* 

RBS-2*,3*, 
4* 


NWREL-4 
RBS-|^*-,3* 


RBS-2*, • 
3*\ 
• 6- 


RBS-3*, 
4* 






, Fomiative 

r 

I. 


CEPM-3*.^6' 
CSE-2,5*, 

. &,io, 

11* 1 
FWL-3,8, 

11, • ; 

14*, 

• is ° 

RBS-2,3* . 
SEDI>-i* 
SWRL-2 >■ 


CEPM-3. 
CSE-2,5, 
6,11* 

FWL-3,8, 
11', 15 

-lRBS-2*,,3* 
^SEDL-1 


CEPM-3, 6' 
FWL-8,11 

RBS-3*,6 

r 

SEDL-1 
SWRE/-2 


<EPM-3 
CSE-2,5*, 
6,11* 

FWL-11 

RBS73*,4* 
SEDL-1. 

• ) 


c ' 


* 


Soimativfe 


CEPM-7 .' ■ 

CSE-2,7*, 

11* 
,FWL-8 
"SEDL-1 


. CEPM-7 

CSE-2>7*, 

11* 
FVJL-8- 
SEDL-1 - 


FWL-8 ^ 
SEDL-1 


CSE-2,7*, 

11* 
SHDL-^. 


- 


« 


General Methodology 
Without Specific 
Type or Purpose 

■ J 


FWL-10^, 
13, 
- 14*, 
16*, 

NWREL-2* 


FMi-10*, 
13, 
16* 

NWREL-2* • 


: 

cFWL-16*:^ 


FWL-IO* 








RBS-5* 
•SWRL-l* 


RBS-5* 
SWRIr-1* ' 


RBS-5* 


SWRL-l* 




• 

ERIC 


> 


"69 

» ^ 




« 

k 







- 








1 - 




f «' 


i 


1 

< 


V i 






* 

Table 5a. Specific PrcxJucts In Cells Of Evaluation Dinension Of Co^eptual Map. 






, (^indicates products vrtiich provide 'specific ho^to training) 






\^ Aspect 9f Evaluation * ' 


) 

\ 




Focus 


Interface 


• 

Administer * 


Determine Constraints 


•> 


V 


CEPM-2* 


CEP^^-4 


CSE-3,6*/10*,ll' 


CSE-4 






CSE-2,4*,6*,7*,11* 


CSE-10 


RVL-15* • 


RBS-4 






• FWI^2*, 3,6,8,11,15* 
16* 


>-HC-3,13 


RBS-2,4* ' 








«BS-2*,3*,4*,5* - 


NWREL-2*,4* 


SEDL-1 
• 








SEDL-1 


^ RBS-"3*,4* 
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SH3L-1 
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Table 6a. 



Specific Products Addressing Categories On Educational Dimension 
Of Conc^tual*Map > y 



Settings \' ' . 




i 

' i 
4 




Schools 


CEPI4-2,3,4>5,6 
RBS-2,3,4,5,6 


C^r2, 3,4,5/6,7/9,10,11 
SEDL-i SlVRL-1 


Professional Organizers 

V 


' EVZ!>4,6,7,8,10,Hvl3,14,15,16) 
. SWRIr-1,2 


SEDL-1,2, 


Ccnniunity ; 


' CEEM-4,5; 


CSE-3,9 ^ FWI/-3 


RBS-3 


Family 


•f 




1 


Contents 








Pblicy 
Resources 


CEPM^4,5,6, 
NWRIr-4 

CEPM-2 


CSE-3,4,9.,10 
RBS-2, 3,4,6 

CSE-5,9,10 , 


FWL-2,3, 
^WREL-4 > 


Programs/Curricula' 


CEEM-2,3,5 
• FWL-2,3>8,14 


CSE-2,3,4,5,6,7,9 
RBS-2, 5,6 . 


,10,11 


Products ^ • 


FWL-8,11,14,15 


SEDL-1,2 


'sWRL-2 


People 

Gei)eral {no . specific 
content) 


CSE-6,9 

FWL-4, 6, 7,10,16 


fMr-13,14 
NWREL-2 


RBS-4 

swrI^ 


• 

Levels 








Societal 

« « , 








Institutional * 

J ' ■. 

_ Instructional * 


CEPf-t-2,4,5,6 
RBS-3 ■ 

' ' CEPM-3,4,5 
RB9-2,6 . 


CSE-4,5,9,10 
•CSE-4,5,9;iO 


NWRL-4 

/ 

FWL-2,14 


Learner 


CEPNt-3 
. RBS-6 


CSE-3,6 


FWL-2,13,14 

' . \ 


•General (no specific ' 
level focus) 

V 


CSE-2,7,11 ' 

NWREL-2 

SWRL-1,2 


FWL-3, 4,6,7, 8,10, 
RBS-3, 4, 5 


11,15,16 
SEDL-1,2 



